Methods and apparatus for generating clinical reports

ABSTRACT

Techniques for documenting a clinical procedure involve transcribing audio data comprising audio of one or more clinical personnel speaking while performing the clinical procedure. Examples of applicable clinical procedures include sterile procedures such as surgical procedures, as well as non-sterile procedures such as those conventionally involving a core code reporter. The transcribed audio data may be analyzed to identify relevant information for documenting the clinical procedure, and a text report including the relevant information documenting the clinical procedure may be automatically generated.

BACKGROUND OF INVENTION

1. Field of Invention

The techniques described herein are directed generally to the field ofclinical documentation, and more particularly to techniques forfacilitating the creation of clinical procedure reports.

2. Description of the Related Art

Clinical documentation is an important process in the healthcareindustry. Most healthcare institutions maintain a longitudinal medicalrecord (e.g., spanning multiple observations or treatments over time)for each of their patients, documenting, for example, the patient'shistory, encounters with clinical staff within the institution,treatment received, and/or plans for future treatment. Suchdocumentation facilitates maintaining continuity of care for the patientacross multiple encounters with various clinicians over time. Inaddition, when an institution's medical records for large numbers ofpatients are considered in the aggregate, the information containedtherein can be useful for educating clinicians as to treatment efficacyand best practices, for internal auditing within the institution, forquality assurance, etc.

One exemplary type of clinical documentation that may be included in apatient's medical record is a clinical procedure report documenting aclinical procedure performed on the patient. For example, one type ofclinical procedure is a surgical procedure. Typically, a surgeon whoperforms a procedure on a patient later creates a report (often referredto as an “operative report”) that documents the procedure in textualform. The operative report may relate information such as the reason(s)for the procedure, the preoperative and postoperative diagnoses, actionsthat were taken and events (including complications) that occurredduring the procedure, observations that were made during the procedure,indications for future treatment, etc. To save time, surgeonsconventionally dictate these operative reports by speaking into arecording device, and the speech is then transcribed (e.g., by a medicaltranscriptionist) to create the text report.

SUMMARY OF INVENTION

One type of embodiment is directed to a method comprising transcribingaudio data comprising audio of one or more clinical personnel speakingwhile performing a surgical procedure; analyzing the transcribed audiodata, using at least one processor, to identify relevant information fordocumenting the surgical procedure; and automatically generating a textreport including the relevant information documenting the surgicalprocedure.

Another type of embodiment is directed to apparatus comprising at leastone processor; and at least one memory storing processor-executableinstructions that, when executed by the at least one processor, performa method comprising transcribing audio data comprising audio of one ormore clinical personnel speaking while performing a surgical procedure,analyzing the transcribed audio data to identify relevant informationfor documenting the surgical procedure, and automatically generating atext report including the relevant information documenting the surgicalprocedure.

Another type of embodiment is directed to at least one computer-readablestorage medium encoded with computer-executable instructions that, whenexecuted, perform a method comprising transcribing audio data comprisingaudio of one or more clinical personnel speaking while performing asurgical procedure; analyzing the transcribed audio data to identifyrelevant information for documenting the surgical procedure; andautomatically generating a text report including the relevantinformation documenting the surgical procedure.

Another type of embodiment is directed to a method comprisingtranscribing audio data comprising audio of one or more clinicalpersonnel speaking while performing a sterile procedure; analyzing thetranscribed audio data, using at least one processor, to identifyrelevant information for documenting the sterile procedure; andautomatically generating a text report including the relevantinformation documenting the sterile procedure.

Another type of embodiment is directed to apparatus comprising at leastone processor; and at least one memory storing processor-executableinstructions that, when executed by the at least one processor, performa method comprising transcribing audio data comprising audio of one ormore clinical personnel speaking while performing a sterile procedure,analyzing the transcribed audio data to identify relevant informationfor documenting the sterile procedure, and automatically generating atext report including the relevant information documenting the sterileprocedure.

Another type of embodiment is directed to at least one computer-readablestorage medium encoded with computer-executable instructions that, whenexecuted, perform a method comprising transcribing audio data comprisingaudio of one or more clinical personnel speaking while performing asterile procedure; analyzing the transcribed audio data to identifyrelevant information for documenting the sterile procedure; andautomatically generating a text report including the relevantinformation documenting the sterile procedure.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIGS. 1A, 1B and 1C illustrate an example of an operative report thatmay be generated in accordance with some embodiments of the presentinvention;

FIG. 2 is a block diagram illustrating an exemplary operatingenvironment for a system in accordance with some embodiments of thepresent invention;

FIG. 3 is a flowchart illustrating an exemplary method for generating areport in accordance with some embodiments of the present invention; and

FIG. 4 is a block diagram illustrating an exemplary computer system onwhich aspects of the present invention may be implemented.

DETAILED DESCRIPTION

Shown in FIG. 1 is an example of an operative report. Exemplaryoperative report 100 was created to document an appendectomy procedure,which is a particular type of surgical procedure. This is just a singleexample provided for illustrative purposes; it should be appreciatedthat operative reports may be created to document any suitableprocedure(s), may take any suitable form and may include any suitablecontent, as aspects of the invention are not limited to any particularoperative report or type of operative report.

Typically, an operative report is divided into a number of sectionsdelineated by section headings, as illustrated in FIG. 1. The inclusionof some of the sections may be required by one or more applicablestandards that specify certain items that must be documented in anoperative report for a particular type of procedure, for a particularinstitution (e.g., a hospital) in which the procedure is performed,and/or for operative reports in general. Other sections of the operativereport in some cases may be included by preference of the surgeon orother clinician who creates the report, or may be pertinent for aparticular patient, a particular procedure performed, etc. One widelyadopted set of standards for operative report content in the UnitedStates is promulgated by the Joint Commission on Accreditation ofHealthcare Organizations (JCAHO). Another notable set of standards ispromulgated by Health Level Seven (HL7). Other standards are availableas well, including standards created and/or applied at a more local orinstitutional level. Any of these standards may be revised at variouspoints in time, and new standards may be developed as the healthcareindustry evolves. It should be appreciated that techniques disclosedherein are not limited to any particular standard for report content,although some embodiments may allow for conformance to particularstandards.

Example operative report 100 begins in FIG. 1A with sections 102 settingforth the patient's diagnoses related to the surgical procedure. Thesection heading “PREOPERATIVE DIAGNOSIS” introduces the statement of thediagnosis that was determined for the patient before the surgicalprocedure was performed—in this case, “perforated appendicitis.” Thesection heading “POSTOPERATIVE DIAGNOSIS” introduces the statement ofthe diagnosis given to the patient after the procedure was completed. Inthis case, the preoperative and postoperative diagnoses are the same;however, in some cases the diagnosis or set of diagnoses may change as aresult of the procedure. For example, clinical observations during theprocedure may give rise to a change in diagnosis, the addition of a newdiagnosis, and/or elimination of a previous diagnosis.

Example operative report 100 continues with section 104 documenting thename(s) of the procedure(s) performed during the operation. In thiscase, two procedures were performed—an appendectomy, and lysis ofadhesions. Section 106 then states the name(s) of one or more of theclinical personnel involved in performing the surgical procedure. Thesurgeon is the primary physician in charge of performing the surgicalprocedure, and is often the person who dictates or otherwise completesthe operative report. The assistant may be another physician or anon-physician clinician (e.g., a medical student or a physicianassistant) who assists the surgeon in performing the surgical procedure,but is not in charge of the surgical procedure. In this case, thepost-nominal initials “PA-C” indicate that Venus Williams is a certifiedphysician assistant, who is licensed to practice medicine under thesupervision of a physician. The anesthesiologist is the physicianresponsible for administering and/or monitoring anesthesia to sedate thepatient during the surgical procedure. In some cases, there may be morethan one surgeon, assistant and/or anesthesiologist involved in thesurgical procedure who may be listed in the operative report. Not alloperative reports include a listing of surgeon, assistant andanesthesiologist; some procedures may not involve one or more of thesepersonnel, and some reports may not list one or more of these personneleven though they were involved in the procedure. In addition, otherclinical personnel may also be involved in performing the surgicalprocedure (e.g., nurses, residents, consulting physicians and/or otherclinical staff), and any of these personnel may or may not be listed inthe operative report. In some cases, the clinical personnel to be listedin the operative report may be determined by an applicable standard forreport content, such as a national or institutional standard, and/or bythe preferences of the clinician creating and/or approving the report.Some operative reports may also list the name or other identification ofthe patient; although this may not be necessary, for example, if thepatient's identity is clear from the inclusion of the operative reportin the patient's medical file or “chart,” or from metadata associatedwith the report in electronic form. Aspects of the invention describedherein are not limited to the listing of any particular personnel in anoperative report, although some embodiments may allow for suchinformation to be included.

Section 108 of example operative report 100 documents the type and/ordosage of anesthesia delivered to the patient during the surgicalprocedure. Section 110, with heading “INDICATION FOR OPERATION,” thenrelates the preoperative events and/or observations that led to thesurgical procedure being selected for the patient. Report contentstandards often also suggest or require documentation to establish thepatient's informed consent to the procedure. Exemplary section 110therefore relates how the patient was counseled on the risks, benefitsand alternatives of the procedure, how the patient understood thecounseling and how he or she agreed to have the procedure performed. Insome other cases, such informed consent documentation may be included asa separate section of the operative report, rather than together withthe patient's presenting symptoms, or may not be included at all. Itshould be appreciated that aspects of the invention are not limited tothe inclusion of any particular information in an operative report,although some embodiments provide techniques for the inclusion ofrequired (e.g., by an institutional or other standard) or otherwiseappropriate content, e.g., for quality assurance purposes.

Example operative report 100 continues in FIG. 1B with section 112, withheading “OPERATIVE FINDINGS.” This section relates observations thatwere made during the surgical procedure, some of which may be relevantto the patient's postoperative diagnosis, to decisions that were madeduring performance of the procedure, and/or to indications for thepatient's further treatment. Example operative report 100 then continueswith a number of sections, some of which may be required by anapplicable report content standard, some of which may be includedroutinely by the clinician creating the report, and/or some of which maybe appropriate for the particular procedure being performed. Section 114documents any tissue that was grafted into the patient, while section116 documents any tissue specimens that were removed from the patient.Section 118 documents a drain that was placed in the patient to removefluid from the patient's abscess. Section 120 documents the estimatedamount of blood that the patient lost during the procedure. Section 122documents any measures that were taken to prevent deep vein thrombosis,which can result from reduced and/or altered blood flow caused bysurgery, by prolonged bed rest after surgery, etc. Section 124 documentsany antibiotics that were administered to the patient, e.g., to preventinfection of the surgical wounds. Section 126 documents anycomplications that occurred during the procedure. Section 128 documentsthe sponge, needle and instrument count, e.g., to confirm that no suchitems were accidentally left inside the patient's body. It should beappreciated that aspects of the present invention are not limited to theinclusion of any of the foregoing sections in an operative report; any,all or none of these sections may be included, and other appropriatesections not mentioned above may be included, as appropriate for aparticular procedure, institution or clinician, and/or as required byany applicable report content standard.

Example operative report 100 concludes in FIG. 1C with a section 130describing the surgical procedure itself. This may be the main narrativeof the operative report supplied, e.g., by the surgeon or other primaryphysician responsible for the operation. Section 130 describes in detailthe actions that were performed during the procedure, the observationsthat were made, and events that occurred during the procedure. Thissection may also include some of the physician's thoughts about thepatient's prognosis and recommendations for further treatment, althoughsuch information may alternatively be documented in a separate sectionor may not be included in the report. Typically, the section of theoperative report narrating the surgical procedure is the section mostleft to the discretion of the physician and/or to the particularities ofthe patient and/or of the procedure performed, although applicablereport content standards may also specify certain types of informationthat should be included in the narrative. A physician typically will tryto narrate all information that may be useful to know for futuretreatment or advising of the patient, as well as information that may beuseful as evidence that proper procedure was followed and that bestefforts were made with regard to the patient's ongoing health. It shouldbe appreciated, however, that aspects of the invention are not limitedto any particular content in the description of a surgical procedure,and that techniques described herein may be used to produce narrationsof any suitable content.

During a surgical procedure, the clinical personnel involved oftenorally communicate a great deal of information to each other, e.g., tohelp them perform the procedure correctly and efficiently together as ateam. For instance, the surgeon often narrates his or her actions outloud as he or she performs them, which may serve to alert the otherpersonnel to what is currently happening in the procedure, allowing themto effectively prepare and assist. For example, the surgeon may sayaloud, “I'm now lysing the omental adhesion,” as he or she begins toperform the associated action. Other personnel may also narrate out loudas they perform their own actions. Clinical personnel may also narratetheir observations of events, complications and/or conditions during theprocedure, such as, “Blood pressure is dropping,” or “I don't see theappendix,” etc. Also, clinical personnel often mention out loud theinstruments used during the surgical procedure. In one example, thesurgeon may ask for the “scalpel,” and a nurse may reply, “scalpel,” ashe or she passes the scalpel to the surgeon. In addition to suchinteractions during the procedure, nurses or other clinical personneloften also make an oral inventory of the instruments and/or othermaterials present in the operating room before and/or after the surgicalprocedure is performed.

The inventor has recognized that much of the information typicallyincluded in an operative report may be gleaned from oral communicationthat occurs during the surgical procedure that the report is created todocument. Yet, conventionally, none of that oral communication is put touse toward documenting the procedure. Instead, physicians conventionallymust wait until sometime after the procedure is completed, and then mustattempt to recall everything that happened and was observed, whiledictating or otherwise creating the operative report from scratch. Theinventor has appreciated that this can be a tedious procedure forphysicians (for whom time can be very valuable), and also can give riseto errors and/or omissions in the documentation due to the limitationsof human memory. The inventor has recognized and appreciated that bycapturing and utilizing the oral communication that already takes placeduring surgical procedures, operative reports may be automaticallygenerated in a way that may both save clinicians valuable time andeffort, and also increase the quality of documentation of surgicalprocedures, which may provide numerous benefits. For example, moredetailed and more accurate clinical documentation can enhance theefficiency of billing, promote quality assurance initiatives, decreaseliability, increase available data for medical research, training anddecision support, etc.

Accordingly, some embodiments described herein relate to techniques forautomatically generating a report (e.g., a text report) for documentinga surgical procedure, using audio of one or more clinical personnelspeaking while performing the surgical procedure. In some embodiments,speech recognition may be performed on the audio recorded during thesurgical procedure, and the recognized speech may then be analyzed toidentify and extract relevant information to include in the operativereport.

As discussed further below, techniques described herein may also beapplied in contexts other than those limited to surgery. For example,some embodiments relate more generally to documenting sterile procedures(which may be surgical or non-surgical), which may require a sterileenvironment and/or may especially require the clinician's hands and/orgloves to be sterile while performing the procedure. For suchprocedures, a clinician may not be able to carry a speech recordingdevice by hand, and/or may not be able to access a computer terminal orother typing or writing tool to document the procedure in real time. Oneexample of such a sterile procedure is a percutaneous endoscopyprocedure, in which an endoscope is used to internally examine a bodycavity of the patient. Such percutaneous endoscopies may involve one ormore incisions to insert one or more scopes into a normally closed bodycavity. Other examples of sterile procedures include any of variousprocedures that may be performed on a patient who is kept in isolationdue to immunodeficiency or contagious disease. Clinicians providing careto such patients may be required to sterilize their hands upon enteringand/or exiting the isolation room.

Other embodiments relate to documenting non-sterile procedures thatinvolve real-time narration and/or documentation of the procedure whilerequiring the clinician to have his or her hands unencumbered to performthe procedure. One example of this type of procedure is in the field ofpathology, in which a pathologist may narrate his or her findings whiledissecting or otherwise analyzing a body, organ, tissue sample or thelike. Another example is an endoscopy that is performed non-surgicallyusing a natural opening of the human body (e.g., the mouth or nose), toexamine a body tract of the patient. Yet another example is an attemptto resuscitate a patient during a type of emergency involving detaileddocumentation by a core code recorder.

The aspects of the present invention described herein can be implementedin any of numerous ways, and are not limited to any particularimplementation techniques. Thus, while examples of specificimplementation techniques are described below, it should be appreciatedthat the examples are provided merely for purposes of illustration, andthat other implementations are possible.

One illustrative application for the techniques described herein is foruse in a system for facilitating the creation of clinical procedurereports. An exemplary operating environment for such a system isillustrated in FIG. 2. The exemplary operating environment includes aprocedure documenting system 200, which may be implemented in anysuitable form, as aspects of the present invention are not limited inthis respect. For example, system 200 may be implemented as a singlestand-alone machine, or may be implemented by multiple distributedmachines that share processing tasks in any suitable manner. System 200may be implemented as one or more computers; an example of a suitablecomputer is described below. In some embodiments, system 200 may includeone or more tangible, non-transitory computer-readable storage devicesstoring processor-executable instructions, and one or more processorsthat execute the processor-executable instructions to perform thefunctions described herein. The storage devices may be implemented ascomputer-readable storage media encoded with the processor-executableinstructions; examples of suitable computer-readable storage media arediscussed below.

As depicted, system 200 includes an automatic speech recognition (ASR)engine 202, a fact extraction component 204, and a report generationcomponent 206. Each of these processing components of system 200 may beimplemented in software, hardware, or a combination of software andhardware. Components implemented in software may comprise sets ofprocessor-executable instructions that may be executed by the one ormore processors of system 200 to perform the functionality describedherein. Each of ASR engine 202, fact extraction component 204 and reportgeneration component 206 may be implemented as a separate component ofsystem 200 (e.g., implemented by hardware and/or software code that isindependent and performs dedicated functions of the component), or anycombination of these components may be integrated into a singlecomponent or a set of distributed components (e.g., hardware and/orsoftware code that performs two or more of the functions describedherein may be integrated, the performance of shared code may bedistributed among two or more hardware modules, etc.). In addition, anyone of ASR engine 202, fact extraction component 204 and reportgeneration component 206 may be implemented as a set of multiplesoftware and/or hardware components. Although the example operatingenvironment of FIG. 2 depicts ASR engine 202, fact extraction component204 and report generation component 206 implemented together on system200, this is only an example; in other examples, any or all of thecomponents may be implemented on one or more separate machines, or partsof any or all of the components may be implemented across multiplemachines in a distributed fashion and/or in various combinations. Itshould be understood that any such component depicted in FIG. 2 is notlimited to any particular software and/or hardware implementation and/orconfiguration.

ASR component 202 may receive speech data via any suitable communicationmedium or media (e.g., local and/or network connection(s) that mayinclude wired and/or wireless connection(s)) from recording component210. Recording component 210 may be implemented in any suitable way torecord audio data from one or more microphones 212 or other audiodetecting and/or receiving devices. References to microphone(s) as usedherein should be interpreted to include any suitable device(s) capableof receiving and transducing audio signals into a form that can bestored by recording component 210 and/or processed by ASR engine 202, asaspects of the invention are not limited to any particular type of audioreceiving device. Recording component 210 may be implemented inhardware, software, or a combination of hardware and software, and mayinclude dedicated recording circuitry and/or one or more processorsprogrammed to record (e.g., store) audio received from microphone(s) 212in a form usable by ASR engine 202 and/or by transcriptionist 230.Alternatively, audio data may be received by ASR engine 202 directlyfrom microphone(s) 212 for processing. The audio data may be representedin any suitable form, such as an analog or digital waveform or asequence of acoustic feature vectors, and may be encoded in any suitabledata format, as aspects of the invention are not limited in thisrespect.

In some embodiments, recording component 210 may have a local connectionto microphone(s) 212, such as via one or more wires or cables capable oftransmitting an audio signal. Alternatively, recording component 210 mayhave a wireless and/or network connection to microphone(s) 212, or mayhave wired connection(s) to some microphone(s) 212 and non-wiredconnection(s) to other microphone(s) 212. Microphone(s) 212 may belocated in a procedure room 280, and recording component 210 may belocally connected and located nearby, e.g., in a machine housing withinprocedure room 280, within a wall of procedure room 280, or in anadjacent room. Alternatively, in some embodiments recording component210 may be integrated with ASR engine 202, on system 200 or in aseparate machine or device, and recording component 210 may receivetransduced audio from microphone(s) 212 remotely via any suitablenetwork connection or other communication medium or media. In some otherembodiments, ASR engine 202 may be integrated locally with recordingcomponent 210 within or near procedure room 280, or system 200 with allits components may even be local to procedure room 280. It should beappreciated that any component or components of the operatingenvironment illustrated in FIG. 2 may be implemented locally or remotelyin any suitable combination, as aspects of the invention are not limitedin this respect.

As illustrated in FIG. 2, one or more clinical personnel 220 may bepresent in procedure room 280, performing a clinical procedure onpatient 222. When the procedure is a surgical procedure (one type ofsterile procedure), procedure room 280 may be an operating room (OR),and clinical personnel 220 may include one or more surgeons, one or moreanesthesiologists, one or more other physicians, one or more assistants,one or more residents, one or more nurses, and/or any other appropriatepersonnel participating in performing and/or supporting the surgicalprocedure. When the procedure is, for example, a pathology procedure(one type of non-sterile clinical procedure), procedure room 280 may bea pathology laboratory, and clinical personnel 220 may include one ormore pathologists, and/or any appropriate assistants or other personnelparticipating in performing and/or supporting the pathology procedure.In this case, the depiction of “patient” 222 may represent a cadaver forautopsy, or one or more organs or tissue samples taken from a patientfor pathology analysis. In general, when the procedure is a non-sterileprocedure, procedure room 280 may be any suitable location; for examplean ambulance, an emergency room or other hospital room in whichresuscitation is performed on a patient 222 experiencing cardiac arrest.

In some embodiments, microphone(s) 212 may be installed within procedureroom 280 in a relatively permanent manner, such that they remain infixed positions during, between and across multiple successiveprocedures performed in procedure room 280. For example, in someembodiments microphone(s) 212 may be affixed to the ceiling and/or wallsof procedure room 280, or may be attached to light fixtures and/or otherequipment or machinery in fixed locations within procedure room 280. Insome embodiments, multiple microphones 212 may be installed withinprocedure room 280 such that audio can be collected from sources in allparts of the room, or in a suitable portion of the room to collectuseful audio for documenting the procedure. Alternatively oradditionally, in some embodiments microphone(s) 212 may be installed inparticular locations and/or orientations within procedure room 280 basedon the normal positions of particular clinical personnel duringprocedures. For example, one or more microphones 212 may be installednear the operating table facing the position in which the surgeonnormally stands, and/or one or more microphones 212 may be installednear the anesthesia machines facing the anesthesiologist. In otherembodiments, microphone(s) 212 may be worn by individual clinicalpersonnel in such a way that their speech is captured by their ownmicrophone(s) in a consistent fashion as they move around procedure room280. As used herein, the reference to microphones being “worn” byclinical personnel refers to attaching the microphones to a portion ofthe body, to clothing, and/or to any other suitable accessory carried onthe body, in a way that does not involve carrying the microphone in thehand. However, in some embodiments, one or more microphones mayalternatively or additionally be carried by hand. Such microphones wornand/or carried by clinical personnel may be used instead of microphonesinstalled in procedure room 280 itself, or in addition to othermicrophones installed in procedure room 280 in a fixed manner.

As clinical personnel 220 are performing the clinical procedure onpatient 222, they may communicate orally to each other through speech.As discussed above, this oral communication may include narration ofactions being performed and/or drugs or other substances beingadministered in connection with the procedure, acknowledgment of eventsoccurring during the procedure, observations of complications occurringand/or conditions presenting in the patient, inventory of instrumentsand/or other materials present and/or used in the procedure, and/orother information relevant for documenting the clinical procedure. Insome embodiments, some or all of this speech may be captured as one ormore audio signals and stored by recording component 210 as audio data.As discussed below, such audio data may then be transcribed manually bya human transcriptionist 230 and/or automatically by ASR engine 202.Alternatively, as discussed above, audio data from microphone(s) 212 maybe processed directly by ASR engine 202, without being handled by aseparate recording component 210. In yet other embodiments, oralcommunication occurring in procedure room 280 may be transcribed in realtime by a human such as a stenographer, without requiring microphone(s)212, recording component 210 or ASR engine 202. Thus, it should beappreciated that in some embodiments, audio data representing oralcommunication may be transcribed in any suitable way, as someembodiments are not limited in this respect.

In some embodiments, the audio may be stored (e.g., by recordingcomponent 210 in some embodiments) and/or processed as a single streamor signal, without consideration of which individual was speaking ateach point in time. In other embodiments, the audio may be stored and/orprocessed as separate signals, channels or files for differentindividual speakers from among clinical personnel 220, and/or fordifferent groups of speakers among clinical personnel 220. For instance,audio of the primary surgeon speaking may be stored and/or processedseparately from audio of the anesthesiologist speaking, and audio ofnurses or other non-physicians speaking may be stored and/or processedtogether with each other but separately from the physicians' audio. Eachseparate set of audio data may then be tagged with an identification ofthe speaker(s) represented by that data, e.g., through metadataassociated with the audio. It should be appreciated that audio data maybe separated or grouped in any suitable manner, as aspects of theinvention are not limited in this respect. In addition, it should beappreciated that audio data, though separated to isolate the speech of aparticular individual or group of individuals, may still contain audiopicked up by the microphone(s) when other individuals were speaking,which may be viewed as a form of background noise for that particularseparated signal.

In some embodiments, separate audio data may be stored and/or processedfor the audio recorded by each microphone present in procedure room 280.When individual microphones are worn by or directed at particularindividual speakers, this may serve to isolate those speakers' speech toa feasible degree. In some embodiments, grouped audio data from multiplemicrophones or audio data from a single microphone that received speechfrom multiple speakers may be labeled or otherwise tagged to designateportions of the audio data that were spoken by particular speakers.Speaker identification may be performed in any suitable way, as aspectsof the invention are not limited in this respect. For example, speakersmay be identified based on which microphone was closest to the speaker(e.g., which microphone picked up the speaker's speech the loudest orwith greatest amplitude) during that portion of speech. In someembodiments, any of various multichannel signal processing techniques,such as beamforming, may be used to focus channels of the audio data onindividual speakers. Alternatively or additionally, speakers may beidentified based on any suitable automatic voice recognition techniquesusing acoustic qualities of individual speakers' voices, and/or based onpeculiarities of word usage, accent, and/or any other suitabletechnique. However, in some embodiments no speaker identification may beperformed at all.

In some embodiments, a transcriptionist 230 may receive the audio datarecorded by recording component 210, and may transcribe it into atextual representation of the oral communication from the clinicalprocedure. Transcriptionist 230 may be any human who listens to theaudio data and writes or types what was spoken into one or more textdocuments. In some embodiments, transcriptionist 230 may be specificallytrained in the field of medical transcription, and may be well-versed inmedical terminology. When transcriptionist 230 has completed thetranscription of the oral communication into a textual representation,the resulting transcription may in some embodiments be transmitted tosystem 200 or to any other suitable location (e.g., to a storagelocation accessible to system 200). Specifically, in some embodimentsthe text transcription may be received from transcriptionist 230 bysystem 200 for processing by fact extraction component 204. Exemplaryfunctionality of fact extraction component 204 is described below.

In some other embodiments, the audio data recorded by recordingcomponent 210 may be received, at system 200 or at any other suitablelocation, by ASR engine 202. In some embodiments, ASR engine 202 maythen process the audio data to determine what was spoken. Suchprocessing may involve any suitable automatic speech recognitiontechnique, as aspects of the present invention are not limited in thisrespect. In some embodiments, the audio recording may be automaticallytranscribed (e.g., converted to a textual representation), while inother embodiments, words identified directly from the audio recordingmay be represented in a data format other than text, or abstractconcepts may be identified instead of words. Examples of furtherprocessing are described below with reference to a transcription that isa textual representation of the oral communication; however, it shouldbe appreciated that similar processing may be performed on otherrepresentations of the oral communication as discussed above. When atextual representation is produced, in some embodiments it may bereviewed by a human (e.g., a transcriptionist) for accuracy, while inother embodiments the output of ASR engine 202 may be accepted asaccurate without human review. As discussed above, some embodiments arenot limited to any particular method for transcribing audio data; oralcommunication from a clinical procedure may be transcribed manually by ahuman transcriptionist, automatically by ASR, or semiautomatically byhuman editing of a draft transcription produced by ASR. Transcriptionsproduced by ASR engine 202 and/or by transcriptionist 230 may be encodedor otherwise represented as data in any suitable form, as aspects of theinvention are not limited in this respect.

In some embodiments, ASR engine 202 may make use of a lexicon ofclinical terms (which may be part of, or in addition to, another moregeneral speech recognition lexicon) while determining the sequence ofwords that were spoken in the oral communication during the clinicalprocedure. However, aspects of the invention are not limited to the useof a lexicon, or any particular type of lexicon, for ASR. When used, theclinical lexicon in some embodiments may be linked to a clinicallanguage understanding ontology or other knowledge representation modelutilized by fact extraction component 204, such that ASR engine 202 canproduce a text transcription containing terms in a form understandableto fact extraction component 204. In some embodiments, a more generalspeech recognition lexicon may also be shared between ASR engine 202 andfact extraction component 204. However, in other embodiments, ASR engine202 may not have any lexicon developed to be in common with factextraction component 204. In some embodiments, a lexicon used by ASRengine 202 may be linked to a different type of clinical knowledgerepresentation model, such as one not designed or used for languageunderstanding. It should be appreciated that any lexicon used by ASRengine 202 and/or by fact extraction component 204 may be implementedand/or represented as data in any suitable way, as aspects of theinvention are not limited in this respect.

As discussed above, in some embodiments the audio data received by ASRengine 202 may be configured such that portions of the oralcommunication spoken by different speakers during the clinical procedurecan be distinguished from each other. For example, audio portions spokenby different speakers may be stored as separate audio files or receivedas separate channels, or portions of the same audio file spoken bydifferent speakers may be labeled, tagged or otherwise marked toidentify which speaker was speaking. In some embodiments, ASR engine 202may make use of this information to improve the speech recognitionprocess. For example, ASR engine 202 may apply different acoustic modelsand/or language models to the recognition of speech from differentspeakers. In some embodiments, such speech recognition models may bespeaker-specific, with each model being trained to recognize the speechof a particular speaker. In other embodiments, speech recognition modelsmay be specific to particular types of speakers. For example, acousticmodels may be specific to gender, and/or to any other suitablecharacteristic tending to produce predictable acoustic differences inspeech. In some embodiments, language models may be specific tospeakers' job categories, to reflect the tendency for personnel withdifferent roles in the clinical procedure to use different words and/orphrases. For example, a different language model may be used for thesurgeon than for the anesthesiologist, and another different languagemodel may be used for nurses and/or other non-physicians. Alternativelyor additionally, language models may be specific to particularprocedures or categories of procedures. For example, the languagemodel(s) used for an appendectomy procedure may be different from thelanguage model(s) used for a hip replacement or an open heart surgery.

Such diversification of speech recognition models may be employed in anysuitable combination, as aspects of the invention are not limited inthis respect. In one example, each individual speaker may have a set ofdifferent speaker-specific models for different types of procedures. Inanother example, acoustic models may be speaker specific while languagemodels may only be specific to job and/or procedure category. In yetanother example, a number of different models with various levels ofspecificity may be utilized for physicians, while fewer and more genericmodels may be used for nurses and/or other non-physicians. Any othersuitable criteria for diversifying speech recognition models may also beemployed. For instance, language model(s) utilized for a currentclinical procedure may be adapted based on the names of the clinicalpersonnel performing the current procedure, the name of the patientand/or any relevant information in the patient's medical history. Forexample, data on the patient's gender may be used to bias the languagemodel away from names of organs or other words specific to the oppositegender. Data on symptoms, diagnoses and/or other medical historyinformation specific to the current patient may also be used to bias thelanguage model. However, in some other embodiments, speech recognitionmodels may not be specific at all to speaker, job category, procedurecategory, patient characteristics and/or any other criteria.

In some embodiments, ASR engine 202 may produce a text transcriptionthat includes or is otherwise associated with timing informationdetailing the points in time at which various portions of the oralcommunication were spoken during the clinical procedure. Such timinginformation in some embodiments may be encoded as metadata such astimestamps associated with the transcription, and/or may be embedded aslabels in the transcription itself, or any other suitable technique maybe used for associating timing information with such a transcription.Similarly, in some embodiments a transcription produced bytranscriptionist 230 may also have associated timing informationindicating the time during the clinical procedure at which portions ofthe oral communication were spoken. Such timing information may becaptured in any suitable way. For example, in some embodimentstranscriptionist 230 may use an interface that allows him or her tomanually associate particular portions of the text transcription withparticular portions of the audio recording, or an electronic system usedby transcriptionist 230 may automatically associate timing informationwith the transcription with reference to the portion of the audio databeing played back as the transcriptionist enters text into the system.Timing information associated with a transcription produced bytranscriptionist 230 or ASR engine 202 may be encoded and/or otherwiserepresented as data in any suitable form, as aspects of the inventionare not limited in this respect. In addition, timing information may beincluded at any suitable level of detail, including multiple levels, asaspects of the invention are not limited in this respect. For example,timestamps may be generated for the transcription at the level oftransitions between the speech of different speakers within the audiodata, at the level of significant pauses in the recorded speech, at thelevel of utterances, sentences, phrases, words, syllables, individualphonemes and/or any other suitable interval. In another example,timestamps may be included at fixed intervals of time during the audiorecording (e.g., every second, or any other suitable time interval), andeach timestamp may be aligned with whatever word or other element in thetext transcription was said at the time represented by the timestamp.

In some embodiments, monitoring equipment 252 may also be present inprocedure room 280 and may, for example, collect physiological data frompatient 222 during the clinical procedure. For example, monitoringequipment 252 may monitor physiological data such as heart rate, bloodpressure, respiratory rate, oxygen saturation, body temperature, and/orany other suitable physiological data (which may vary as appropriatebased upon circumstances such as characteristics of the patient and/orthe procedure being performed). Alternatively or additionally,monitoring equipment 252 may make other measurements such as the amountof drugs, fluids, etc., that are administered to the patient during theclinical procedure. However, aspects of the invention are not limited tothe use of any particular monitoring data. In some embodiments,monitoring data produced by monitoring equipment 252 may be collectedand/or stored by monitoring component 250, which may be implementedlocally to procedure room 280, remotely at system 200, or in any othersuitable configuration. Monitoring component 250 may be implemented inany suitable form, including as hardware, software, or a combination ofhardware and software, and may include dedicated recording circuitryand/or one or more processors programmed to record (e.g., store)monitoring data received from monitoring equipment 252. The monitoringdata may be represented in any suitable form and may be encoded in anysuitable data format, as aspects of the invention are not limited inthis respect. In some embodiments, monitoring component 250 may alsoassociate timing information with the recorded monitoring data. Forexample, if monitoring data is sampled at discrete intervals in time,the sampling rate may be included, along with the starting time and/orending time of the monitoring, in the monitoring data compiled bymonitoring component 250, to allow for calculation of the time at whichparticular events occurred in the monitoring data. Monitoring component250 may also, in some embodiments, label significant events in themonitoring data with particular time indices, such as points in time atwhich the patient's heart rate dropped below a predetermined threshold,for example.

In some embodiments, the text transcription of the oral communicationtranscribed by ASR engine 202 and/or by transcriptionist 230, andoptionally the monitoring data collected by monitoring component 250,may be processed by fact extraction component 204. Such data may betransmitted or otherwise made accessible to fact extraction component204 via any suitable form of local and/or network connection(s), and/orby communication between hardware and/or software modules of a commondevice upon which the components involved may be implemented. In someembodiments, fact extraction component 204 may perform processing (e.g.,by operation of the one or more processors of system 200 to implementinstructions stored in a memory of system 200) to extract one or moreclinical facts from the transcription. As one example, if thetranscription includes a statement, “I'm now lysing the adhesion,”spoken by the surgeon, fact extraction component 204 may extract aclinical fact such as, “adhesion was lysed,” from this portion of thetranscription.

Any suitable technique(s) for extracting clinical facts from the texttranscription may be used, as aspects of the present invention are notlimited in this respect. In some embodiments, fact extraction component204 may be implemented using techniques such as those described in U.S.Pat. No. 7,493,253, entitled “Conceptual World Representation NaturalLanguage Understanding System and Method.” This patent is incorporatedherein by reference in its entirety. Such a fact extraction component204 may make use of a formal ontology linked to a lexicon of clinicalterms. The formal ontology may be implemented as a relational database,or in any other suitable form, and may represent semantic conceptsrelevant to the clinical domain, as well as linguistic concepts relatedto ways the semantic concepts may be expressed in natural language.

In some embodiments, concepts in a formal ontology used by factextraction component 204 may be linked to a lexicon of clinical termsand/or codes, such that each clinical term and each code is linked to atleast one concept in the formal ontology. In some embodiments, thelexicon may include the standard clinical terms and/or codes used by theinstitution in which fact extraction component 204 is applied. In someembodiments, the lexicon may also include variations on the standardclinical terms and/or additional clinical terms used by clinicalpersonnel in oral communication during clinical procedures. Suchadditional clinical terms may be linked, along with their correspondingstandard clinical terms, to the appropriate shared concepts within theformal ontology. For example, the standard term “cardiac arrest” as wellas other orally used terms such as “crashing” and “coding” may all belinked to the same abstract concept in the formal ontology—a conceptrepresenting cessation of heart beat.

In some embodiments, a formal ontology used by fact extraction component204 may also represent various types of relationships between theconcepts represented. One type of relationship between two concepts maybe a parent-child relationship, in which the child concept sharesattributes of the parent concept as well as additional attributes. Insome embodiments, any other type(s) of relationship useful to theprocess of clinical documentation may also be represented in the formalontology. For example, one type of relationship may be a complicationrelationship. In one example of a complication relationship, a conceptlinked to the term “pneumothorax” (collapsed lung) may have arelationship of “is-complication-of” to the concept linked to the term“pacemaker implantation” (a type of surgical procedure). Thisrelationship may represent the knowledge that pneumothorax is a knowncomplication of pacemaker implantations. Any number and type(s) ofconcept relationships may be included in such a formal ontology, asaspects of the present invention are not limited in this respect.

In some embodiments, automatic extraction of clinical facts from a texttranscription may involve parsing the transcription to identify clinicalterms that are represented in the lexicon of fact extraction component204. Concepts in the formal ontology linked to the clinical terms thatappear in the transcription may then be identified, and conceptrelationships in the formal ontology may be traced to identify furtherrelevant concepts. Through these relationships, as well as thelinguistic knowledge represented in the formal ontology, one or moreclinical facts may be extracted. In some embodiments, relationshipsbetween concepts in the formal ontology may also allow fact extractioncomponent 204 to automatically extract facts containing clinical termsthat were not explicitly stated in the oral communication that wastranscribed.

For instance, in one example, oral communication during a pacemakerimplantation procedure may contain a statement from one of the clinicalpersonnel: “The patient's lung is collapsing.” Fact extraction component204 may search the lexicon for recognized words such as “lung” and“collapsing.” These words may be linked together to the concept“pneumothorax” in the ontology, from which fact extraction component 204may extract the fact that a pneumothorax has occurred. Further, theconcept “pneumothorax” may be linked in the ontology to the concept“pacemaker implantation” via the relationship “is-complication-of.” Bytracing this relationship, fact extraction component 204 may extract themore specific fact that a pneumothorax has occurred as a complication ofthe pacemaker implantation procedure. Then, as discussed below, thisfact may be incorporated into an automatically generated operativereport, e.g., by listing “Pneumothorax” in a section with heading“COMPLICATIONS.”

It should be appreciated that the foregoing descriptions are provided byway of example only, and that any suitable technique(s) may be used forextracting a set of one or more clinical facts from a text transcriptionof oral communication during a clinical procedure, as aspects of thepresent invention are not limited to any particular fact extractiontechnique. For instance, it should be appreciated that fact extractioncomponent 204 is not limited to the use of an ontology, as other formsof knowledge representation models, including statistical models and/orrule-based models, may also be applicable. The knowledge representationmodel may also be represented as data in any suitable format, and may bestored in any suitable location, such as in a memory of system 200accessible by fact extraction component 204, as aspects of the inventionare not limited in this respect. In addition, a knowledge representationmodel such as an ontology used by fact extraction component 204 may beconstructed in any suitable way, as aspects of the invention are notlimited in this respect.

For instance, in some embodiments a knowledge representation model maybe constructed manually by one or more human developers with access toexpert knowledge about clinical procedures, their constituent steps,associated diagnoses, potential complications, appropriate observationsand/or clinical findings, and/or any other relevant information. Inother embodiments, a knowledge representation model may be generatedautomatically, for example through statistical analysis of pastoperative reports, of transcriptions of oral communication duringclinical procedures, and/or of other medical documents. Thus, in someembodiments, fact extraction component 204 may have access to a data set270 of medical literature and/or other documents such as past operativereports and/or transcriptions. In some embodiments, past reports,transcriptions and/or other text documents may be marked up (e.g., by ahuman) with labels indicating the nature of the relevance of particularstatements in the text to the clinical procedure(s) to which the textrelates. A statistical knowledge representation model may then betrained to form associations based on their prevalence within anaggregate set of multiple marked up documents. For example, if“pneumothorax” is labeled as a “complication” in a large enoughproportion of operative reports documenting pacemaker implantationprocedures, a statistical knowledge representation model may generateand store a concept relationship that “pneumothorax is-complication-ofpacemaker implantation.” In some embodiments, automatically generatedand hard coded (e.g., by a human developer) concepts and/orrelationships may both be included in a knowledge representation modelused by fact extraction component 204.

As discussed above, it should be appreciated that aspects of theinvention are not limited to any particular technique(s) forconstructing knowledge representation models. Examples of suitabletechniques include those disclosed in the following

-   Gómez-Pérez, A., and Manzano-Macho, D. (2005). An overview of    methods and tools for ontology learning from texts. Knowledge    Engineering Review 19, p. 187-212.-   Cimiano, P., and Staab, S. (2005). Learning concept hierarchies from    text with a guided hierarchical clustering algorithm. In C. Biemann    and G. Paas (eds.), Proceedings of the ICML 2005 Workshop on    Learning and Extending Lexical Ontologies with Machine Learning    Methods, Bonn, Germany.-   Fan, J., Ferrucci, D., Gondek, D., and Kalyanpur, A. (2010).    PRISMATIC: Inducing Knowledge from a Lange Scale Lexicalized    Relation Resource. NAACL Workshop on Formalisms and Methodology for    Learning by Reading.-   Welty, C., Fan, J., Gondek, D. and Schlaikjer, A. (2010). Large    scale relation detection. NAACL Workshop on Formalisms and    Methodology for Learning by Reading.

Each of the foregoing publications is incorporated herein by referencein its entirety.

In another example, fact extraction component 204 may utilize astatistical knowledge representation model based on entity detectionand/or tracking techniques, such as those disclosed in: Florian, R.,Hassan, H., Ittycheriah, A., Jing, H., Kambhatla, N., Luo, X., Nicolov,N., and Roukos, S. (2004). A Statistical Model for Multilingual EntityDetection and Tracking. Proceedings of the Human Language TechnologiesConference 2004 (HLT-NAACL'04). This publication is incorporated hereinby reference in its entirety.

For example, in some embodiments, a list of fact types of interest forgenerating clinical procedure reports may be defined, e.g., by adeveloper of fact extraction component 204. Such fact types may include,for example, diagnoses, procedures, procedure steps, observations,medications, problems, and/or any other suitable fact types. It shouldbe appreciated that any suitable list of fact types may be utilized, andmay or may not include any of the fact types listed above, as aspects ofthe invention are not limited in this respect. In some embodiments,spans of text in a set of sample transcriptions of oral communicationduring clinical procedures may be labeled (e.g., by a human) withappropriate fact types from the list. For example, if a sampletranscription contains the statement, “I'm now lysing the omentaladhesion,” a human annotator may label the span, “lysing the omentaladhesion,” with the “procedure step” fact type label. A statisticalmodel may then be trained on the corpus of labeled sample transcriptionsto detect and/or track such fact types as semantic entities, using knownentity detection and/or tracking techniques.

In some embodiments, a statistical model may also be trained toassociate fact types extracted from new transcriptions with particularfacts to be used in generating a clinical procedure report. For example,in some embodiments, the human annotator may label “lysing the omentaladhesion” not only with the “procedure step” fact type label, but alsowith a label indicating the particular procedure to which this stepbelongs (e.g., “lysis of adhesions”) and/or the particular procedurestep fact that the text indicates (e.g., “lysis of omental adhesion”).In such embodiments, for example, a single statistical model may betrained to detect and/or track specific particular facts as individualentities. However, in other embodiments, the process of detecting and/ortracking fact types as entities may be separated from the process ofrelating detected fact types to particular facts. For example, in someembodiments, a separate statistical model may be trained on differenttraining data to take as input spans such as “lysing the omentaladhesion,” previously labeled with the “procedure step” fact type label,and relate them to the particular procedure “lysis of adhesions” and/orto the particular fact “lysis of omental adhesion.” In otherembodiments, such a relation model may be hard-coded and/or otherwiserule-based, while the entity detection and/or tracking model used toextract fact types from text spans may be trained statistically.

It should be appreciated that the foregoing are merely examples, andthat fact extraction component 204 may be implemented in any suitableway and/or form, as aspects of the invention are not limited in thisrespect.

In some embodiments, different full knowledge representation models maybe constructed for different types of clinical procedures, or differentpartial knowledge representation models specific to particularprocedures may be accessed selectively in conjunction with a genericknowledge representation model. In other embodiments, a single knowledgerepresentation model may be constructed with concepts and relationshipsadequate to address all clinical procedures for which the knowledgerepresentation model is used, and the relationships stored may determinethe likelihood with which certain concepts are activated in the contextof a particular procedure.

In some embodiments, a knowledge representation model or portion of aknowledge representation model may be made specific to a particular typeof procedure by training it on transcriptions, operative reports and/orother medical documents specific to that type of procedure. In oneexample, operative report 100 documenting an appendectomy procedure,once approved by a responsible clinician, may be used to train aknowledge representation model on concepts and associations relevant toappendectomies. For example, if the statements, “omental adhesion,”“partial closed loop small-bowel obstruction,” and “ischemic bowel” arelabeled in section 112 of the appendectomy report as “observations,” theknowledge representation model may learn that these concepts arerelevant clinical observations for appendectomy procedures. Mark-up ofsection 130 of example operative report 100 may also aid in training theknowledge representation model on the procedural steps involved in anappendectomy and the order in which they are performed: “anesthesia,”followed by “prepped and draped,” then “incision,” “suctioning,” etc. Asan alternative or in addition to the process of (e.g., manual) mark-upof documents for model training, section headings within the documentsmay also be used in building associations. For example, the term “Foleycatheter” in section 118 may be learned as a relevant “drain” for anappendectomy procedure, because of the section heading, “DRAINS.”

In some embodiments, a knowledge representation model used by factextraction component 204 may also be trained on (and/or may be linked toa separate model component trained on) transcriptions of oralcommunication spoken during past clinical procedures. Such modeltraining may allow fact extraction component 204 to extract clinicalfacts from oral communication transcriptions despite the varied ways inwhich clinical personnel may speak about such information during aprocedure, which may be quite different from the type of language usedin operative reports and/or in medical literature. For example, whenspeaking to each other about a patient who is experiencing excessivebleeding during a surgical procedure, clinical personnel may speak suchvaried utterances as, “Blood pressure is dropping,” “We've got a bleederhere,” “We need to ligate quickly,” “I'm clamping,” and, “Where's itcoming from?” In some embodiments, any of such similar statements in atranscription of oral communication from a past clinical procedure maybe labeled (e.g., manually) as indicative of “excessive bleeding” or“hemorrhage.” Further, “excessive bleeding” and/or “hemorrhage” may alsobe labeled as “complications” of the surgical procedure corresponding tothe transcription. Such labeled text may then be used to train factextraction component 204 (e.g., by training the model(s) utilized byfact extraction component 204) to extract a clinical fact related to“hemorrhage” as a “complication” from such statements in futuretranscriptions for that clinical procedure, and/or from combinations ofsuch statements with high likelihood of “hemorrhage”-related labels inthe training data.

In some embodiments, rather than receive a complete transcription of theentire oral communication from a current clinical procedure from ASRengine 202 before performing any fact extraction, fact extractioncomponent 204 may perform some of its processing concurrently with ASRengine 202 on a current clinical procedure transcription. For instance,the inventor has recognized that typical words and/or phrases spoken byclinical personnel 220 may change in predictable ways between differentstages of a clinical procedure. To take advantage of thispredictability, in some embodiments different language models may betrained for different stages of a clinical procedure, and/or a languagemodel may be trained to apply different statistics to different stagesof the clinical procedure. In some embodiments, ASR engine 202 may beinformed by fact extraction component 204 as to what is the currentstage of the clinical procedure for each portion of the audio data, andmay apply its language models to the speech recognition accordingly. Inone example, ASR engine 202 may initially transcribe a first portion ofthe audio data using a generic language model, and may pass the partialtranscription to fact extraction component 204. Fact extractioncomponent 204 may then extract one or more clinical facts from the firstportion of the audio data, and may determine from the extracted fact(s)what stage of the procedure was being performed. This information may berelayed back to ASR engine 202, which may then transcribe the nextportion of the audio data using the language model(s) appropriate forthat stage of the procedure. In another example, ASR engine 202 may bydefault apply a “preparation” language model to the first portion of theaudio data for, e.g., a surgical procedure, based on the assumption thatmost surgeries begin with a preparation stage. Fact extraction component204 may then extract clinical facts from each portion of thetranscription as it is provided by ASR engine 202, and may alert ASRengine 202 when sufficient facts have been extracted to indicate that atransition, e.g., from the preparation stage to the incision stage, hasoccurred or will soon occur. ASR engine 202 may then switch to an“incision” language model for recognizing subsequent portions of theoral communication. However, it should be appreciated that certainaspects of the invention are not limited to any particular languagemodeling techniques, and some embodiments may not include the use ofmultiple stage-specific language models.

As discussed above, audio data (e.g., collected by recording component210) may in some embodiments be labeled or otherwise marked withidentification of which speaker was talking during various portions ofthe oral communication. In some embodiments, the transcription of theoral communication produced by ASR engine 202 and/or by transcriptionist230 may also have portions marked by speaker, as their correspondingportions in the audio data were. In some embodiments, fact extractioncomponent 204 may use this information to inform its extraction ofclinical facts from the transcription. For example, a mention of adosage amount by the anesthesiologist may give rise to extraction of afact that that amount of anesthetic was administered to the patientduring the clinical procedure, while a similar utterance by differentpersonnel may not give rise to extraction of such a fact.

In some embodiments, fact extraction component 204 may also receivemonitoring data collected by monitoring component 250, and may use thismonitoring data to extract clinical facts for documenting the clinicalprocedure, independently of the transcribed oral communication. Forexample, monitoring data collected by monitoring component 250 mayprovide the measure of 100 mL of estimated blood loss documented insection 120 of example operative report 100. In addition, in someembodiments monitoring data collected by monitoring component 250 mayinform the extraction of clinical facts from the transcription of theoral communication. In some embodiments, the oral communication and themonitoring data may be integrated by synchronizing the timinginformation associated with each. For example, if the monitoring dataindicates the occurrence of a significant event (e.g., a sudden drop inblood pressure, heart rate or other physiological measure) at aparticular time index during the clinical procedure, fact extractioncomponent 204 can then search within the vicinity of that time index inthe transcription and/or the audio data for clinical facts that may beextracted to explain the event reflected in the monitoring data. In somecases, certain statements in the transcription may have differentconnotations and/or different significance when viewed in light ofconcurrently recorded monitoring data than when viewed in isolation. Insome embodiments, timing information associated with the transcription,with the audio data and/or with the monitoring data may also be used byfact extraction component 204 to determine the duration of the procedureand/or of any of its constituent parts or steps. Such objectivedetermination of duration information may in itself represent asignificant step forward in clinical procedure documentation practices,which conventionally must rely upon the memory of the physician fordocumentation of the approximate durations of individual parts of theprocedure.

In some embodiments, fact extraction component 204 may also have accessto a data set 260 of patient history records, from which it may accessthe medical history for the patient upon which the current clinicalprocedure was performed. In some cases, much of the general informationneeded for documenting the clinical procedure may be available directlyfrom the patient's medical chart and/or electronic health record, orfrom extracting facts from past narrative documentation within thepatient's chart. Such information may include the patient's name,preoperative diagnosis, indication for operation, drugs administeredpreoperatively, and/or any other suitable information documented priorto the clinical procedure. In addition, information from the patient'srecords may also be used by fact extraction component 204 in someembodiments to inform the clinical facts that it extracts from thetranscription of the oral communication spoken during the clinicalprocedure. For example, certain clinical findings, complications,comorbidities, etc., may be more likely during the clinical procedurebased on the patient's known presenting symptoms, preoperative diagnosisand/or medical history, and fact extraction component 204 may interpretstatements in the oral communication accordingly. As discussed above,the patient's history records or facts extracted from such records mayalso be used by ASR engine 202 to select and/or modify the languagemodel(s) it uses in transcribing the audio data.

As should be appreciated from the foregoing, the extraction of clinicalfacts from a transcription of oral communication spoken during aclinical procedure in some embodiments may involve distinguishingrelevant information from non-relevant information in the transcription.For example, a surgeon's conversation with his assistant during asurgical procedure about what he did for fun over the past weekend isnot likely to be relevant for documenting the surgical procedure. Italso is not likely to correspond to clinical concepts in a knowledgerepresentation model used by fact extraction component 204, andtherefore not likely to be extracted from the transcription as aclinical fact. In addition, in some embodiments, having extractedmultiple clinical facts from the transcription of the oralcommunication, from the monitoring data, and/or from patient historyrecords 260, fact extraction component 204 may then filter the extractedclinical facts to determine which facts are relevant for documenting theclinical procedure. For example, the knowledge representation model mayindicate that a certain amount of bleeding is routinely expected incertain types of surgical procedures, but is a notable complication inother types of surgical procedures. Thus, for the types of proceduresfor which bleeding is routine, certain extracted facts related tobleeding may be determined not to be relevant to documenting thatprocedure, and fact extraction component 204 in some embodiments maytherefore filter those facts out of the set of facts that will be usedto create the operative report. However, in other embodiments, extractedclinical facts may be retained despite having less relevance to thecurrent clinical procedure, to err on the side of inclusiveness ratherthan omission. Further, in some embodiments fact extraction component204 may make determinations as to which extracted facts are relevant andtherefore to be retained for documentation based on one or moreapplicable report content standards, which may suggest and/or requirecertain information to be documented in a clinical procedure report, asdiscussed above.

In some embodiments, a set of relevant clinical facts extracted by factextraction component 204 may be added as discrete structured data itemsto a structured data repository such as an electronic health record(EHR) for the patient. However, in other embodiments, relevant extractedclinical facts may be alternatively or additionally received from factextraction component 204 by report generation component 206. Such datamay be transmitted to report generation component 206 via any suitableform of local and/or network connection(s), and/or by communicationbetween hardware and/or software modules of a common device upon whichthe components involved may be implemented. In some embodiments, reportgeneration component 206 may perform processing (e.g., by operation ofone or more processors of system 200 to execute instructions stored in amemory of system 200) to automatically generate a text report (such asan operative report) from the relevant clinical facts extracted by factextraction component 204 from the transcription of the oralcommunication spoken during the clinical procedure. In some embodiments,the text report may document the clinical procedure at least partiallyin narrative natural language format, e.g., detailing multiple steps ofthe procedure through semantically connected sentences having anarrative flow. An example of this type of narrative natural languageformat is illustrated in sections 110, 112 and 130 of example operativereport 100 in FIG. 1; however, this is just one example, and aspects ofthe invention are not limited to this particular form of operativereport.

In some embodiments, a report generation model used by report generationcomponent 206 to automatically generate text reports may be constructedmanually, e.g., by one or more human experts. In other embodiments, thereport generation model may be trained statistically on the same or asimilar set of past clinical procedure reports (e.g., accessed frommedical document data set 270) as is used to train the knowledgerepresentation model used by fact extraction component 204. For example,in some embodiments the past text reports that are marked up withclinical facts to associate with particular portions of text may also bemarked up by a syntactic and/or grammatical parser, to train the reportgeneration model on the way clinical facts are expressed in naturallanguage sentences in the context of other surrounding sentences. Whilefact extraction component 204 may use text training data to learn how toextract discrete structured clinical facts from free-form oralcommunication, report generation component 206 may use similar trainingdata to, in a sense, “work backward” from the discrete structured factsto generate a natural language narration typical of an operative report.In some embodiments, report generation component 206 may also use one ormore report generation models that are specific to generating reportsfor particular clinicians who will ultimately review and/or authorizethe reports. Such report generation models may be trained, for example,on past clinical procedure reports that were written, dictated, editedor otherwise reviewed and/or authorized by the corresponding clinician,and/or may be trained and/or retrained as the clinician continues toauthorize new reports, as discussed below. In some embodiments, a reportgeneration model may be partially statistically trained and partiallymanually constructed.

In some embodiments, report generation component 206 may also accesspatient history records 260 for information to be included in the textreport documenting the clinical procedure. In some cases, certainsections and/or portions of the text report may simply be copied frompast narrative reports that are already part of the patient's medicalrecord. For example, section 110 of example operative report 100,relating the indication for operation, may already have been created indocumenting a past clinical encounter with the patient, and in that casemay be copied directly from such past documentation. In some cases,texts portions of smaller sizes, such as individual sentences orphrases, may be copied from past reports from the patient's historyand/or from other stored medical documents. For example, the reportgeneration model may be trained that a particular clinician usuallyexpresses a particular clinical fact using a particular sentence orphrase. In other cases, data from patient history records 260 may beapplied to the report generation model as discrete clinical facts (e.g.,stored as such in patient history records 260, or extracted from patienthistory records 260 by fact extraction component 204), and the reportgeneration model may create natural language text corresponding to thefacts to populate the text report.

In some embodiments, report generation component 206 may begin a processof generating a text report to document a clinical procedure, byidentifying a set of report sections appropriate and/or required fordocumenting the particular clinical procedure at hand. This may be donein any suitable way. In some embodiments, an applicable report contentstandard may dictate a set of sections to be included in a reportdocumenting a particular type of clinical procedure. In someembodiments, report generation component 206 may be programmed with adefault set of sections for any suitable criteria. For example, adefault set of sections may be specified for clinical procedures ingeneral, or different sets of sections may be specified for surgicalprocedures than for pathology procedures or other types of procedures,or different sets of sections may be specified for different types ofsurgical procedures (such as appendectomies), or different sets ofsections may be specified for reports produced for different clinicians,or for different institutions, or for any other suitable categorizationof clinical procedure reports. As an example, operative report 100depicted in FIGS. 1A-1C includes the set of sections {“PREOPERATIVEDIAGNOSIS,” “POSTOPERATIVE DIAGNOSIS,” “NAME OF PROCEDURE,” “SURGEON,”“ASSISTANT,” “ANESTHESIOLOGIST,” “ANESTHESIA,” “INDICATION FOROPERATION,” “OPERATIVE FINDINGS,” “GRAFTS,” “SPECIMENS,” “DRAINS,”“ESTIMATED BLOOD LOSS,” “DEEP VEIN THROMBOSIS PROPHYLAXIS,”“ANTIBIOTICS,” “COMPLICATIONS,” “SPONGE, NEEDLE, AND INSTRUMENT COUNT,”“DESCRIPTION”}

In some embodiments, report generation component 206 may determine whichfacts of the set of clinical facts to be included in the report shouldbe included in which sections of the set of report sections identifiedfor the report, using any suitable technique(s). In some embodiments,report generation component 206 may be programmed with a set of rulesthat associate particular clinical facts, or particular types ofclinical facts, with particular sections of a clinical procedure report.In some embodiments, such rules may be represented as part of aknowledge representation model such as an ontology. In some embodiments,report generation component 206 may be trained statistically to assignclinical facts to report sections. For example, in some embodiments,clinical facts may be extracted from a training set of past clinicalprocedure reports, and the report sections from which those clinicalfacts were extracted may be used to train report generation component206 to determine the best report sections in which to include newclinical facts for new reports.

In some embodiments, report generation component 206 may be configuredto determine an order in which to include multiple clinical facts withina particular section of a clinical procedure report. For example, insome embodiments, report generation component 206 may make use of thetiming information associated with particular clinical facts by othersystem components, and may order the clinical facts in temporal orderwithin a report section. Alternatively or additionally, in someembodiments report generation component 206 may make use of a knowledgerepresentation model such as an ontology, or other specified rulesindicating that certain clinical facts can only occur after otherclinical facts. For instance, in one example, a rule may specify that acomplication of a surgical procedure can only occur after the surgicalprocedure has been initiated. It should be appreciated that reportgeneration component 206 may order clinical facts in any suitable way,as aspects of the invention are not limited in this respect.

In some embodiments, report generation component 206 may then generatenatural language sentences to represent the ordered clinical factswithin the set of report sections for the clinical procedure report.Natural language generation (NLG) may be performed in any suitable way,as aspects of the invention are not limited in this respect. In someembodiments, known NLG techniques may be adapted to produce clinicalprocedure reports. Exemplary NLG techniques are disclosed in: Reiter,E., and Dale, R. (2000). Building Natural Language Generation Systems.Cambridge: Cambridge University Press. This publication is incorporatedherein by reference in its entirety. Other NLG techniques may also beused. In some embodiments, report generation component 206 may beprogrammed with specified individual sentences to use to representparticular individual clinical facts. Alternatively or additionally, insome embodiments report generation component 206 may be programmed tocombine multiple facts into the same sentence. This may be done in anysuitable way. For example, in some embodiments, report generationcomponent 206 may be programmed with multiple-fact sentence templateswith slots to be filled by particular clinical facts. In one example,report generation component 206 may be programmed with a sentencetemplate, “The patient is a(n) X-year-old Y,” where X and Y are slots tobe filled by the patient's age fact and gender fact (male or female),respectively. Report generation component 206 may be programmed to use“a” or “an” depending on whether the fact inserted in slot X begins witha consonant or a vowel. In another example, report generation component206 may be programmed with a sentence template, “A(n) X-catheter wasinserted in the Y,” where X is filled with a catheter type, such as“Pigtail,” “Foley,” “French Foley,” or “triple lumen,” and Y is filledwith a body part, such as “urinary bladder,” or “femoral artery.” Insome embodiments, report generation component 206 may be programmed tocombine facts of similar types into the same sentence. For example,instead of producing separate sentences, “White blood cell count was7.6,” “Hemoglobin was 11.4,” and “Platelet count was 37,000,” reportgeneration component 206 may be programmed to list the bloodwork factstogether in a single sentence, such as, “White blood cell count was 7.6,hemoglobin was 11.4, and platelet count was 37,000.” In someembodiments, complex sentences may be generated using a discourseplanner to generate discourse specifications from clinical facts, and asurface realizer to generate sentences from discourse specifications.For example, a surface realizer could generate the sentence, “The aortawas dissected using a scalpel,” from the discourse specification:{(process: dissect), (actee: aorta), (voice: passive), (tense: past),(means: scalpel)}. Many other examples are possible, and it should beappreciated that aspects of the invention are not limited to anyparticular implementation of NLG technique(s).

In some embodiments, a text report, once generated by report generationcomponent 206, may be stored in patent history records 260, as part ofthe patient's medical record, and/or with medical literature/documents270, for use in future research and/or in training and/or retrainingmodels for the fact extraction and/or report generation processes.However, in other embodiments, a draft report may be provided for reviewby a user 242 prior to finalization and storage of the text report. User242 may be one of clinical personnel 220, such as the primary physicianresponsible for the clinical procedure being documented, or may be aperson other than a clinician, such as a report editing specialist. Insome embodiments, both a non-clinician editor/reviewer and a physicianmay review and/or edit the text report, as described below.

In some embodiments, user 242 may review and/or edit the draft reportgenerated by report generation component 206 through a user interface240 provided in connection with system 200. Such a user interface may beimplemented in any suitable way and/or form, as aspects of the inventionare not limited in this respect. In one example, a user interface may beimplemented using techniques disclosed in U.S. patent application Ser.No. 13/030,981, filed Feb. 18, 2011, entitled “Methods and Apparatus forLinking Extracted Clinical Facts to Text.” This patent application isincorporated herein by reference in its entirety. In some embodiments,user interface 240 may be presented on a visual display. However, theuser interface is not limited to a graphical user interface, as otherways of providing data to users from system 200 may be used. Forexample, in some embodiments, audio indicators and/or text-to-speechsynthesis may be conveyed to user 242, e.g., via a speaker. It should beappreciated that any type of user interface may be provided inconnection with report review and/or other related processes, as aspectsof the invention are not limited in this respect.

In some embodiments, the draft report generated by report generationcomponent 206 may be displayed or otherwise presented to user 242, whomay be given the opportunity to edit the report. In some embodiments, alinkage may be maintained between each extracted clinical fact and theportion(s) of the transcription of the oral communication from whichthat fact was extracted, and/or between each corresponding portion ofthe draft text report and the portion(s) of the transcription of theoral communication from which the fact giving rise to that portion ofthe draft text report was extracted. For example, if a factcorresponding to “appendix not found” is extracted from statements inthe transcription including, “I don't see the appendix,” and in turngives rise to generation of text in the narrative of the reportincluding, “The appendix was searched for. The distal portion could notbe found” (as illustrated in section 130 of example operative report100), then a linkage may be maintained between each of the extractedfact, the corresponding portion of the transcription, and thecorresponding portion of the text report. In some embodiments, whileuser 242 is reviewing the draft report via user interface 240 to reportgeneration component 206, the system may provide one or more indicatorsto user 242 of the different linkages between various portions of thedraft report, the different extracted facts from which they weregenerated, and/or the portions of the transcription from which the factswere extracted. Such indicators may be visual indicators, audioindicators, or any other suitable type of indicators, as aspects of theinvention are not limited in this respect. In some embodiments, suchlinkage indicators may enhance the ability of user 242 to review theextracted facts and/or the corresponding portions of the draft reportfor accuracy, with reference to the specific parts of the oralcommunication that generated them. While some embodiments providelinkage information for each extracted fact, it should be appreciatedthat aspects of the invention relating to providing linkage informationare not so limited, as linkage information may be provided for one orany subset of the extracted facts.

In some embodiments, user 242 may use user interface 240 to make changesto the draft report generated by report generation component 206. Forexample, user 242 may add text to the draft report and/or add facts tothe set of clinical facts extracted by fact extraction component 204,may delete text from the draft report and/or delete facts from the setof clinical facts, and/or may modify text and/or facts displayed withthe draft report. In some embodiments, such user edits may be providedas feedback to update and/or to re-train report generation component 206and/or fact extraction component 204, which may result in someembodiments in user-specific fact extraction and/or report generationmodels. In some embodiments, user feedback may be used to adapt themodels for more accurate fact extraction and/or report generation forthe user's current editing session only, while in other embodiments,user-specific adapted models may be stored for future use for that user,and/or may be cumulatively adapted over time. Also, in some embodimentsthe feedback may be used to further modify the draft report and/or theset of extracted clinical facts. If user 242 makes changes to the set ofextracted clinical facts, in some embodiments report generationcomponent 206 may re-generate an updated text report from the new set offacts. The updated text report may then be returned to user 242 forfurther review, or stored as the final version of the text report. Ifuser 242 makes changes to the text of the draft report, in someembodiments fact extraction component 204 may re-extract clinical factsfrom the new report, e.g., for inclusion in a discrete structured datarepository such as the patient's EHR. However, not all embodiments maymake use of user feedback, as aspects of the invention are not limitedin this respect.

In some embodiments, automatically extracted and/or user-suppliedclinical facts may also be automatically reviewed, and automatic alertsmay be provided to user 242 if opportunities are identified for thedocumentation of the clinical procedure to be improved. Such alerts maybe visual alerts, audio alerts, or any other suitable type of alerts, asaspects of the invention are not limited in this respect. For example,in some embodiments fact extraction component 204, report generationcomponent 206 and/or any other suitable component of a proceduredocumenting system may be programmed based on an applicable standardspecifying minimum information to be documented in a clinical procedurereport, examples of which are discussed above. In some embodiments, whenit is determined that additional information should be included in thetext report to comply with one or more applicable standards (e.g., suchas JCAHO or HL7), one or more prompts may be provided to user 242 to addinformation to the text report. Also, in some embodiments, a prompt maybe provided when the system determines that disambiguation is desiredbetween multiple facts that could potentially be extracted from the sameportion of the oral communication during the clinical procedure.Furthermore, user 242 may choose for any suitable reason to addinformation to the text report, for example to supply additional detailsnot captured by the oral communication during the procedure, and/or todocument impressions, ideas and/or plans for future treatment that mayhave occurred to the physician after the procedure was over.

It should be appreciated that techniques described herein may beimplemented in the context of various different workflows, as aspects ofthe invention are not limited in this respect. In some embodiments,review and/or editing of the text report may be performed entirely by aclinician or entirely by a non-clinician reviewer. In other embodiments,a non-clinician reviewer may edit the report in a first pass, and thenthe clinician may further edit and/or finalize the report. In stillother embodiments, the clinician may make a first rough edit of thedraft report, and then a non-clinician reviewer may formalize and/orfinalize the report. In typical circumstances, a clinician such as theprimary physician responsible for the clinical procedure will finalize,authorize and/or otherwise approve the text report, after which it willnot be edited further. However, this is not required, and it should beappreciated that certain aspects of the invention are not limited to anyparticular reviewing and/or editing technique and/or configuration. Onceapproved, in some embodiments the text report documenting the clinicalprocedure may be stored with the patient's medical record, and/or may beincluded in a larger data set such as medical documents 270. In someembodiments, further fact extraction may be performed on the finalizedprocedure report, for purposes such as coding for billing, qualityassurance review, etc.

It should be appreciated from the foregoing that one embodiment of theinvention is directed to a method 300 for automatically generating atext report documenting a clinical procedure, as illustrated in FIG. 3.Method 300 may be performed, for example, by one or more components of aprocedure documenting system 200 such as ASR engine 202, fact extractioncomponent 204 and report generation component 206, although otherimplementations are possible, as method 300 is not limited in thisrespect. Method 300 begins at act 310, at which audio data comprisingaudio of one or more clinical personnel speaking while performing aclinical procedure may be transcribed, e.g., by automatic speechrecognition or by a human transcriptionist. At act 320, the transcribedaudio data may be analyzed to identify relevant information fordocumenting the clinical procedure. Method 300 ends at act 330, at whicha text report including the relevant information documenting theclinical procedure may be automatically generated.

As discussed above, one type of clinical procedure that may bedocumented using techniques described herein is a surgical procedure;however, certain aspects of the present invention are not limited tosurgical contexts. For example, another application for techniquesdescribed herein is in documenting sterile procedures in general. Asurgical procedure may be one example type of sterile procedure. Anotherexemplary type of sterile procedure may be a clinical procedureperformed on a patient in isolation in a hospital or other clinicalinstitution, due to immunodeficiency, infectious disease, and/or anyother suitable reason for isolating the patient. In the case ofimmunodeficiency, efforts are generally made to keep a clinician'shands/gloves (often in addition to other parts of the body and/orclothing) sterile when entering the room in which the patient isisolated. In the case of infectious disease, efforts are generally madeto re-sterilize (or discard) anything worn and/or carried out of thepatient's room after treatment of the patient. In each of these cases,the requirements of a sterile environment and/or hands-free performingof a procedure may make carrying around a hand-held recording devicedifficult and/or dangerous. Accordingly, techniques described herein,for example using sterile fixed microphones installed in the procedureroom, may allow for unencumbered and uncontaminated performance ofsterile procedures with relevant documentation information captured fromoral communication that takes place during the procedure.

As discussed above, another exemplary application for techniquesdescribed herein may be in certain non-sterile clinical procedures suchas resuscitation attempts conventionally requiring a core code reporter.For example, when a patient (e.g., in an ambulance, an emergency room, ahospital room, or any other suitable location) experiences cardiacarrest, attending clinical personnel will typically attempt toresuscitate the patient by “calling a code.” Certain applicablestandards require that an individual known as a “core code reporter”document the exact nature and timing of actions that are taken andevents that occur during the resuscitation attempt. The inventor hasrecognized that techniques described herein may be applied toautomatically generate such documentation from the oral communicationthat typically occurs during such a procedure. This may have the effectof lessening or perhaps even eliminating the need for a core codereporter to stand idly by performing documentation while a life-or-deathemergency procedure is performed.

Another example type of clinical procedure may be a pathology proceduresuch as an autopsy, which is typically documented in a text report suchas an autopsy report. During an autopsy, a pathologist typicallydocuments each of his or her detailed findings and/or observations as heor she methodically dissects and analyzes the deceased's body. Suchfindings may be recorded using speech and a hand-held voice recorder;however, the use of the hand-held recording device may interfere withthe pathologist's use of his or her hands during the autopsy. In somecases, an autopsy may need to be a sterile procedure, and use of ahand-held voice recorder may promote contamination of the sterileenvironment in proximity to the cadaver. By contrast, techniquesdescribed herein may be applied to allow the pathologist's hands toremain free (and in some cases sterile) during the procedure. Inaddition, pathologists conventionally write their own autopsy reports,using the recorded impressions only as a reminder of what to includemanually in the report. By contrast, techniques described herein may beapplied to at least partially automate the process of generating thetext report, saving valuable time and effort for the pathologist. Thepathologist may then be free to extemporize during the pathologyprocedure about his or her observations as they come to mind, withouthaving to be concerned about the order, format or particular language ofthe resulting documentation, which may be generated automatically.

A clinical procedure documentation system in accordance with thetechniques described herein may take any suitable form, as aspects ofthe present invention are not limited in this respect. An illustrativeimplementation of a computer system 400 that may be used in connectionwith some embodiments of the present invention is shown in FIG. 4. Oneor more computer systems such as computer system 400 may be used toimplement any of the functionality described above. The computer system400 may include one or more processors 410 and one or more tangible,non-transitory computer-readable storage media (e.g., volatile storage420 and one or more non-volatile storage media 430, which may be formedof any suitable non-volatile data storage media). The processor 410 maycontrol writing data to and reading data from the volatile storage 420and/or the non-volatile storage device 430 in any suitable manner, asaspects of the present invention are not limited in this respect. Toperform any of the functionality described herein, processor 410 mayexecute one or more instructions stored in one or more computer-readablestorage media (e.g., volatile storage 420), which may serve as tangible,non-transitory computer-readable storage media storing instructions forexecution by the processor 410.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. It should beappreciated that any component or collection of components that performthe functions described above can be generically considered as one ormore controllers that control the above-discussed functions. The one ormore controllers can be implemented in numerous ways, such as withdedicated hardware, or with general purpose hardware (e.g., one or moreprocessors) that is programmed using microcode or software to performthe functions recited above.

In this respect, it should be appreciated that one implementation ofembodiments of the present invention comprises at least onecomputer-readable storage medium (e.g., a computer memory, a floppydisk, a compact disk, a magnetic tape, or other tangible, non-transitorycomputer-readable medium) encoded with a computer program (i.e., aplurality of instructions), which, when executed on one or moreprocessors, performs above-discussed functions of embodiments of thepresent invention. The computer-readable storage medium can betransportable such that the program stored thereon can be loaded ontoany computer resource to implement aspects of the present inventiondiscussed herein. In addition, it should be appreciated that thereference to a computer program which, when executed, performsabove-discussed functions, is not limited to an application programrunning on a host computer. Rather, the term “computer program” is usedherein in a generic sense to reference any type of computer code (e.g.,software or microcode) that can be employed to program one or moreprocessors to implement above-discussed aspects of the presentinvention.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” “having,” “containing,” “involving,” andvariations thereof, is meant to encompass the items listed thereafterand additional items. Use of ordinal terms such as “first,” “second,”“third,” etc., in the claims to modify a claim element does not byitself connote any priority, precedence, or order of one claim elementover another or the temporal order in which acts of a method areperformed. Ordinal terms are used merely as labels to distinguish oneclaim element having a certain name from another element having a samename (but for use of the ordinal term), to distinguish the claimelements.

Having described several embodiments of the invention in detail, variousmodifications and improvements will readily occur to those skilled inthe art. Such modifications and improvements are intended to be withinthe spirit and scope of the invention. Accordingly, the foregoingdescription is by way of example only, and is not intended as limiting.The invention is limited only as defined by the following claims and theequivalents thereto.

What is claimed is:
 1. A method comprising: transcribing audio datacomprising audio of one or more clinical personnel speaking whileperforming a surgical procedure, the audio data comprising audio of afirst clinician speaking to one or more other clinical personnel whileperforming the surgical procedure; analyzing the transcribed audio data,including the transcribed audio of the first clinician speaking to theone or more other clinical personnel while performing the surgicalprocedure, at least in part by automatically extracting one or moreclinical facts from the transcribed audio data using a fact extractioncomponent implemented via at least one processor, to identify relevantinformation for documenting the surgical procedure, wherein analyzingthe transcribed audio data comprises identifying within the transcribedaudio data a present-tense narration by the first clinician stating tothe other clinical personnel that the first clinician is currentlyperforming a particular step of the surgical procedure; automaticallygenerating a text report including the relevant information documentingthe surgical procedure, wherein automatically generating the text reportcomprises automatically transforming the present-tense narration into anon-present-tense text portion in the report, stating that theparticular step of the surgical procedure was performed; and outputtingthe automatically generated text report for review via a user interfaceon an audio and/or visual display device.
 2. The method of claim 1,wherein the audio data comprises audio of the one or more clinicalpersonnel orally identifying one or more surgical instruments used inperforming the surgical procedure.
 3. The method of claim 1, wherein theaudio data comprises audio of the one or more clinical personnel orallyidentifying one or more actions performed during the surgical procedure.4. The method of claim 1, wherein the audio data comprises audio of theone or more clinical personnel orally identifying one or morecomplications occurring during the surgical procedure.
 5. The method ofclaim 1, wherein the surgical procedure is performed on a patient, andwherein the audio data comprises audio of the one or more clinicalpersonnel orally identifying one or more observations of a condition ofthe patient during the surgical procedure.
 6. The method of claim 1,wherein the surgical procedure is performed on a patient, and whereinthe audio data comprises audio of the one or more clinical personnelorally identifying one or more substances administered to the patient inconnection with the surgical procedure.
 7. The method of claim 1,further comprising recording the audio data through one or moremicrophones installed in an operating room in which the surgicalprocedure is performed.
 8. The method of claim 1, further comprisingrecording the audio data through one or more microphones worn by the oneor more clinical personnel.
 9. The method of claim 1, wherein thetranscribing comprises performing automatic speech recognition (ASR) onthe audio data to generate a transcription.
 10. The method of claim 9,wherein the ASR is performed using different speaker-specific models foraudio produced by different speakers from among the one or more clinicalpersonnel.
 11. The method of claim 9, wherein the ASR is performed usinga different language model for the surgical procedure than for adifferent surgical procedure.
 12. The method of claim 9, wherein thesurgical procedure is a first type of surgical procedure, and whereinthe ASR is performed using a language model trained to be specific tothe first type of surgical procedure.
 13. The method of claim 9, furthercomprising allowing a human to verify accuracy of the transcriptionbefore performing the analyzing.
 14. The method of claim 1, wherein theanalyzing is performed using at least one model trained on past surgicalreports, past audio transcriptions and/or medical literature.
 15. Themethod of claim 1, wherein the analyzing is performed using a differentmodel for the surgical procedure than for a different surgicalprocedure.
 16. The method of claim 1, wherein the surgical procedure isa first type of surgical procedure, and wherein the analyzing isperformed using a model trained to be specific to the first type ofsurgical procedure.
 17. The method of claim 1, wherein automaticallyextracting the one or more clinical facts comprises: identifying withinthe transcribed audio data a statement made by the first clinician tothe other clinical personnel while performing the surgical procedure;automatically determining whether information in the statement by thefirst clinician to the other clinical personnel is relevant fordocumenting the surgical procedure; in response to determining that theinformation in the statement by the first clinician to the otherclinical personnel is relevant for documenting the surgical procedure,automatically designating the relevant information for inclusion indocumentation of the surgical procedure; and in response to determiningthat the information in the statement by the first clinician to theother clinical personnel is not relevant for documenting the surgicalprocedure, disregarding the statement in documenting the surgicalprocedure.
 18. The method of claim 1, wherein the analyzing comprisesidentifying the relevant information based at least in part on rulesregarding information to be documented for an institution responsiblefor the surgical procedure.
 19. The method of claim 1, wherein theanalyzing comprises identifying the relevant information based at leastin part on identifying which portions of the audio were spoken by whichof the one or more clinical personnel.
 20. The method of claim 1,wherein the analyzing comprises determining, from the audio data, timinginformation for at least one step performed during the surgicalprocedure, and including the timing information in the relevantinformation.
 21. The method of claim 1, wherein the analyzing comprisesidentifying the relevant information based at least in part on patientmonitoring data recorded using monitoring equipment during the surgicalprocedure.
 22. The method of claim 21, further comprising synchronizingtiming information from the patient monitoring data with timinginformation from the audio data.
 23. The method of claim 1, wherein thesurgical procedure is performed on a patient, and wherein the analyzingcomprises identifying the relevant information based at least in part onmedical history data for the patient.
 24. The method of claim 1, whereinthe generating is performed using at least one model trained on pastsurgical reports.
 25. The method of claim 24, wherein the at least onemodel is specific to one of the one or more clinical personnel whoauthorizes the text report.
 26. The method of claim 1, wherein thegenerating comprises copying text from at least one past surgical reportthat also included the relevant information.
 27. The method of claim 1,further comprising allowing a human user to make edits to the textreport.
 28. The method of claim 27, further comprising updating at leastone model used in the generating based on the human user's edits to thetext report.
 29. The method of claim 27, further comprising promptingthe human user to add information to the text report to comply with oneor more report content standards.
 30. The method of claim 1, wherein thesurgical procedure is performed on a patient, and wherein the methodfurther comprises including at least part of the relevant information ina discrete structured data repository for the patient.
 31. Apparatuscomprising: at least one processor; and at least one storage mediumstoring processor-executable instructions that, when executed by the atleast one processor, perform a method comprising: transcribing audiodata comprising audio of one or more clinical personnel speaking whileperforming a surgical procedure, the audio data comprising audio of afirst clinician speaking to one or more other clinical personnel whileperforming the surgical procedure; analyzing the transcribed audio data,including the transcribed audio of the first clinician speaking to theone or more other clinical personnel while performing the surgicalprocedure, at least in part by automatically extracting one or moreclinical facts from the transcribed audio data, to identify relevantinformation for documenting the surgical procedure, wherein analyzingthe transcribed audio data comprises identifying within the transcribedaudio data a present-tense narration by the first clinician stating tothe other clinical personnel that the first clinician is currentlyperforming a particular step of the surgical procedure; automaticallygenerating a text report including the relevant information documentingthe surgical procedure, wherein automatically generating the text reportcomprises automatically transforming the present-tense narration into anon-present-tense text portion in the report, stating that theparticular step of the surgical procedure was performed; and outputtingthe automatically generated text report for review via a user interfaceon an audio and/or visual display device.
 32. The apparatus of claim 31,wherein the audio data comprises audio of the one or more clinicalpersonnel orally identifying one or more surgical instruments used inperforming the surgical procedure.
 33. The apparatus of claim 31,wherein the audio data comprises audio of the one or more clinicalpersonnel orally identifying one or more actions performed during thesurgical procedure.
 34. The apparatus of claim 31, wherein the audiodata comprises audio of the one or more clinical personnel orallyidentifying one or more complications occurring during the surgicalprocedure.
 35. The apparatus of claim 31, wherein the surgical procedureis performed on a patient, and wherein the audio data comprises audio ofthe one or more clinical personnel orally identifying one or moreobservations of a condition of the patient during the surgicalprocedure.
 36. The apparatus of claim 31, wherein the surgical procedureis performed on a patient, and wherein the audio data comprises audio ofthe one or more clinical personnel orally identifying one or moresubstances administered to the patient in connection with the surgicalprocedure.
 37. The apparatus of claim 31, wherein the method furthercomprises recording the audio data through one or more microphonesinstalled in an operating room in which the surgical procedure isperformed.
 38. The apparatus of claim 31, wherein the method furthercomprises recording the audio data through one or more microphones wornby the one or more clinical personnel.
 39. The apparatus of claim 31,wherein the transcribing comprises performing automatic speechrecognition (ASR) on the audio data to generate a transcription.
 40. Theapparatus of claim 39, wherein the ASR is performed using differentspeaker-specific models for audio produced by different speakers fromamong the one or more clinical personnel.
 41. The apparatus of claim 39,wherein the ASR is performed using a different language model for thesurgical procedure than for a different surgical procedure.
 42. Theapparatus of claim 39, wherein the surgical procedure is a first type ofsurgical procedure, and wherein the ASR is performed using a languagemodel trained to be specific to the first type of surgical procedure.43. The apparatus of claim 39, wherein the method further comprisesallowing a human to verify accuracy of the transcription beforeperforming the analyzing.
 44. The apparatus of claim 31, wherein theanalyzing is performed using at least one model trained on past surgicalreports, past audio transcriptions and/or medical literature.
 45. Theapparatus of claim 31, wherein the analyzing is performed using adifferent model for the surgical procedure than for a different surgicalprocedure.
 46. The apparatus of claim 31, wherein the surgical procedureis a first type of surgical procedure, and wherein the analyzing isperformed using a model trained to be specific to the first type ofsurgical procedure.
 47. The apparatus of claim 31, wherein automaticallyextracting the one or more clinical facts comprises: identifying withinthe transcribed audio data a statement made by the first clinician tothe other clinical personnel while performing the surgical procedure;automatically determining whether information in the statement by thefirst clinician to the other clinical personnel is relevant fordocumenting the surgical procedure; in response to determining that theinformation in the statement by the first clinician to the otherclinical personnel is relevant for documenting the surgical procedure,automatically designating the relevant information for inclusion indocumentation of the surgical procedure; and in response to determiningthat the information in the statement by the first clinician to theother clinical personnel is not relevant for documenting the surgicalprocedure, disregarding the statement in documenting the surgicalprocedure.
 48. The apparatus of claim 31, wherein the analyzingcomprises identifying the relevant information based at least in part onrules regarding information to be documented for an institutionresponsible for the surgical procedure.
 49. The apparatus of claim 31,wherein the analyzing comprises identifying the relevant informationbased at least in part on identifying which portions of the audio werespoken by which of the one or more clinical personnel.
 50. The apparatusof claim 31, wherein the analyzing comprises determining, from the audiodata, timing information for at least one step performed during thesurgical procedure, and including the timing information in the relevantinformation.
 51. The apparatus of claim 31, wherein the analyzingcomprises identifying the relevant information based at least in part onpatient monitoring data recorded using monitoring equipment during thesurgical procedure.
 52. The apparatus of claim 51, wherein the methodfurther comprises synchronizing timing information from the patientmonitoring data with timing information from the audio data.
 53. Theapparatus of claim 31, wherein the surgical procedure is performed on apatient, and wherein the analyzing comprises identifying the relevantinformation based at least in part on medical history data for thepatient.
 54. The apparatus of claim 31, wherein the generating isperformed using at least one model trained on past surgical reports. 55.The apparatus of claim 54, wherein the at least one model is specific toone of the one or more clinical personnel who authorizes the textreport.
 56. The apparatus of claim 31, wherein the generating comprisescopying text from at least one past surgical report that also includedthe relevant information.
 57. The apparatus of claim 31, wherein themethod further comprises allowing a human user to make edits to the textreport.
 58. The apparatus of claim 57, wherein the method furthercomprises updating at least one model used in the generating based onthe human user's edits to the text report.
 59. The apparatus of claim57, wherein the method further comprises prompting the human user to addinformation to the text report to comply with one or more report contentstandards.
 60. The apparatus of claim 31, wherein the surgical procedureis performed on a patient, and wherein the method further comprisesincluding at least part of the relevant information in a discretestructured data repository for the patient.
 61. At least onenon-transitory computer-readable storage medium encoded withcomputer-executable instructions that, when executed, perform a methodcomprising: transcribing audio data comprising audio of one or moreclinical personnel speaking while performing a surgical procedure, theaudio data comprising audio of a first clinician speaking to one or moreother clinical personnel while performing the surgical procedure;analyzing the transcribed audio data, including the transcribed audio ofthe first clinician speaking to the one or more other clinical personnelwhile performing the surgical procedure, at least in part byautomatically extracting one or more clinical facts from the transcribedaudio data, to identify relevant information for documenting thesurgical procedure, wherein analyzing the transcribed audio datacomprises identifying within the transcribed audio data a present-tensenarration by the first clinician stating to the other clinical personnelthat the first clinician is currently performing a particular step ofthe surgical procedure; automatically generating a text report includingthe relevant information documenting the surgical procedure, whereinautomatically generating the text report comprises automaticallytransforming the present-tense narration into a non-present-tenseportion in the report, stating that the particular step of the surgicalprocedure was performed; and outputting the automatically generated textreport for review via a user interface on an audio and/or visual displaydevice.
 62. The at least one non-transitory computer-readable storagemedium of claim 61, wherein the audio data comprises audio of the one ormore clinical personnel orally identifying one or more surgicalinstruments used in performing the surgical procedure.
 63. The at leastone non-transitory computer-readable storage medium of claim 61, whereinthe audio data comprises audio of the one or more clinical personnelorally identifying one or more actions performed during the surgicalprocedure.
 64. The at least one non-transitory computer-readable storagemedium of claim 61, wherein the audio data comprises audio of the one ormore clinical personnel orally identifying one or more complicationsoccurring during the surgical procedure.
 65. The at least onenon-transitory computer-readable storage medium of claim 61, wherein thesurgical procedure is performed on a patient, and wherein the audio datacomprises audio of the one or more clinical personnel orally identifyingone or more observations of a condition of the patient during thesurgical procedure.
 66. The at least one non-transitorycomputer-readable storage medium of claim 61, wherein the surgicalprocedure is performed on a patient, and wherein the audio datacomprises audio of the one or more clinical personnel orally identifyingone or more substances administered to the patient in connection withthe surgical procedure.
 67. The at least one non-transitorycomputer-readable storage medium of claim 61, wherein the method furthercomprises recording the audio data through one or more microphonesinstalled in an operating room in which the surgical procedure isperformed.
 68. The at least one non-transitory computer-readable storagemedium of claim 61, wherein the method further comprises recording theaudio data through one or more microphones worn by the one or moreclinical personnel.
 69. The at least one non-transitorycomputer-readable storage medium of claim 61, wherein the transcribingcomprises performing automatic speech recognition (ASR) on the audiodata to generate a transcription.
 70. The at least one non-transitorycomputer-readable storage medium of claim 69, wherein the ASR isperformed using different speaker-specific models for audio produced bydifferent speakers from among the one or more clinical personnel. 71.The at least one non-transitory computer-readable storage medium ofclaim 69, wherein the ASR is performed using a different language modelfor the surgical procedure than for a different surgical procedure. 72.The at least one non-transitory computer-readable storage medium ofclaim 69, wherein the surgical procedure is a first type of surgicalprocedure, and wherein the ASR is performed using a language modeltrained to be specific to the first type of surgical procedure.
 73. Theat least one non-transitory computer-readable storage medium of claim69, wherein the method further comprises allowing a human to verifyaccuracy of the transcription before performing the analyzing.
 74. Theat least one non-transitory computer-readable storage medium of claim61, wherein the analyzing is performed using at least one model trainedon past surgical reports, past audio transcriptions and/or medicalliterature.
 75. The at least one non-transitory computer-readablestorage medium of claim 61, wherein the analyzing is performed using adifferent model for the surgical procedure than for a different surgicalprocedure.
 76. The at least one non-transitory computer-readable storagemedium of claim 61, wherein the surgical procedure is a first type ofsurgical procedure, and wherein the analyzing is performed using a modeltrained to be specific to the first type of surgical procedure.
 77. Theat least one non-transitory computer-readable storage medium of claim61, wherein automatically extracting the one or more clinical factscomprises: identifying within the transcribed audio data a statementmade by the first clinician to the other clinical personnel whileperforming the surgical procedure; automatically determining whetherinformation in the statement by the first clinician to the otherclinical personnel is relevant for documenting the surgical procedure;in response to determining that the information in the statement by thefirst clinician to the other clinical personnel is relevant fordocumenting the surgical procedure, automatically designating therelevant information for inclusion in documentation of the surgicalprocedure; and in response to determining that the information in thestatement by the first clinician to the other clinical personnel is notrelevant for documenting the surgical procedure, disregarding thestatement in documenting the surgical procedure.
 78. The at least onenon-transitory computer-readable storage medium of claim 61, wherein theanalyzing comprises identifying the relevant information based at leastin part on rules regarding information to be documented for aninstitution responsible for the surgical procedure.
 79. The at least onenon-transitory computer-readable storage medium of claim 61, wherein theanalyzing comprises identifying the relevant information based at leastin part on identifying which portions of the audio were spoken by whichof the one or more clinical personnel.
 80. The at least onenon-transitory computer-readable storage medium of claim 61, wherein theanalyzing comprises determining, from the audio data, timing informationfor at least one step performed during the surgical procedure, andincluding the timing information in the relevant information.
 81. The atleast one non-transitory computer-readable storage medium of claim 61,wherein the analyzing comprises identifying the relevant informationbased at least in part on patient monitoring data recorded usingmonitoring equipment during the surgical procedure.
 82. The at least onenon-transitory computer-readable storage medium of claim 81, wherein themethod further comprises synchronizing timing information from thepatient monitoring data with timing information from the audio data. 83.The at least one non-transitory computer-readable storage medium ofclaim 61, wherein the surgical procedure is performed on a patient, andwherein the analyzing comprises identifying the relevant informationbased at least in part on medical history data for the patient.
 84. Theat least one non-transitory computer-readable storage medium of claim61, wherein the generating is performed using at least one model trainedon past surgical reports.
 85. The at least one non-transitorycomputer-readable storage medium of claim 84, wherein the at least onemodel is specific to one of the one or more clinical personnel whoauthorizes the text report.
 86. The at least one non-transitorycomputer-readable storage medium of claim 61, wherein the generatingcomprises copying text from at least one past surgical report that alsoincluded the relevant information.
 87. The at least one non-transitorycomputer-readable storage medium of claim 61, wherein the method furthercomprises allowing a human user to make edits to the text report. 88.The at least one non-transitory computer-readable storage medium ofclaim 87, wherein the method further comprises updating at least onemodel used in the generating based on the human user's edits to the textreport.
 89. The at least one non-transitory computer-readable storagemedium of claim 87, wherein the method further comprises prompting thehuman user to add information to the text report to comply with one ormore report content standards.
 90. The at least one non-transitorycomputer-readable storage medium of claim 61, wherein the surgicalprocedure is performed on a patient, and wherein the method furthercomprises including at least part of the relevant information in adiscrete structured data repository for the patient.